{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.</i>\n",
    "\n",
    "<i>Licensed under the MIT License.</i>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "2bd54cdb-aa39-48f9-8bb0-d602135c1b6c"
    }
   },
   "source": [
    "# Quickstart: Web Cam Action Recognition\n",
    "\n",
    "Action recognition is the increasingly popular computer vision task of determining specific actions in a given video.\n",
    "\n",
    "This notebook shows a simple example of loading a pretrained R(2+1)D model for action recognition and using a webcam stream to identify what actions are being performed.\n",
    "\n",
    "For more details about the underlying technology of action recognition, including finetuning, please see our [training introduction notebook](01_training_introduction.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisite for Webcam example \n",
    "This notebook assumes you have **a webcam** connected to your machine.  We use the `ipywebrtc` module to show the webcam widget in the notebook. Currently, the widget works on **Chrome** and **Firefox**. For more details about the widget, please visit `ipywebrtc` [github](https://github.com/maartenbreddels/ipywebrtc) or [documentation](https://ipywebrtc.readthedocs.io/en/latest/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "nbpresent": {
     "id": "89d4b446-c421-488c-8967-8a38e538a9b2"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.7.6 (default, Jan  8 2020, 19:59:22) \n",
      "[GCC 7.3.0] \n",
      "\n",
      "PyTorch 1.2.0 \n",
      "\n",
      "Torch-vision 0.4.0a0 \n",
      "\n",
      "Available devices:\n",
      "0: Tesla K80\n"
     ]
    }
   ],
   "source": [
    "# Regular Python libraries\n",
    "import sys\n",
    "from collections import deque #\n",
    "import io\n",
    "import requests\n",
    "import os\n",
    "from time import sleep, time\n",
    "from threading import Thread\n",
    "from IPython.display import Video\n",
    "\n",
    "# Third party tools\n",
    "import decord #\n",
    "import IPython.display #\n",
    "from ipywebrtc import CameraStream, ImageRecorder\n",
    "from ipywidgets import HBox, HTML, Layout, VBox, Widget, Label\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "import torch\n",
    "import torch.cuda as cuda\n",
    "import torch.nn as nn\n",
    "from torchvision.transforms import Compose\n",
    "\n",
    "# utils_cv\n",
    "sys.path.append(\"../../\")\n",
    "from utils_cv.action_recognition.data import KINETICS, Urls\n",
    "from utils_cv.action_recognition.dataset import get_transforms\n",
    "from utils_cv.action_recognition.model import VideoLearner\n",
    "from utils_cv.action_recognition.references import transforms_video as transforms\n",
    "from utils_cv.common.gpu import system_info, torch_device\n",
    "from utils_cv.common.data import data_path\n",
    "\n",
    "system_info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "nbpresent": {
     "id": "6db47566-8c49-44bb-aa28-c63bf3fb63cd"
    }
   },
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "cf40a98f-414d-4827-8aa8-88ef6b9cc626"
    }
   },
   "source": [
    "## Load Pre-trained Model\n",
    "\n",
    "Load R(2+1)D 34-layer model pre-trained on IG65M or Kinetics400. There are also two versions of the model that we provide by default: an 8-frame model and 32-frame model based on the input clip length. As you'd expect, the 32-frame model is slower than 8-frame model. \n",
    "\n",
    "We'll start by setting some of these parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "nbpresent": {
     "id": "c456a2db-b030-46fc-8cd7-55c133feb354"
    },
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "NUM_FRAMES = 8  # 8 or 32.\n",
    "IM_SCALE = 128  # resize then crop\n",
    "INPUT_SIZE = 112  # input clip size: 3 x NUM_FRAMES x 112 x 112\n",
    "\n",
    "# video sample to download\n",
    "sample_video_url = Urls.webcam_vid\n",
    "\n",
    "# file path to save video sample\n",
    "video_fpath = data_path() / \"sample_video.mp4\"\n",
    "\n",
    "# prediction score threshold\n",
    "SCORE_THRESHOLD = 0.01\n",
    "\n",
    "# Averaging 5 latest clips to make video-level prediction (or smoothing)\n",
    "AVERAGING_SIZE = 5  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we just need to initialize our VideoLearner model and add the parameters we set above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "nbpresent": {
     "id": "c6f4b85d-2f55-4d5a-8edb-5a9b9f8cc8b4"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading r2plus1d_34_8_kinetics model\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using cache found in /home/jiata/.cache/torch/hub/moabitcoin_ig65m-pytorch_master\n"
     ]
    }
   ],
   "source": [
    "learner = VideoLearner(\n",
    "    base_model=\"kinetics\",\n",
    "    sample_length=NUM_FRAMES,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "8cec9405-d125-4681-9bdd-ec2184257de3"
    }
   },
   "source": [
    "## Prepare class names and prediction variables\n",
    "Since we use Kinetics400 model out of the box, we load its class names. The dataset consists of 400 human actions. For example, the first 10 labels are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "nbpresent": {
     "id": "eb1dfefe-dfe4-46e4-8482-9b2bc1d05b89"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['abseiling',\n",
       " 'air drumming',\n",
       " 'answering questions',\n",
       " 'applauding',\n",
       " 'applying cream',\n",
       " 'archery',\n",
       " 'arm wrestling',\n",
       " 'arranging flowers',\n",
       " 'assembling computer',\n",
       " 'auctioning']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "LABELS = KINETICS.class_names\n",
    "LABELS[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "5e7de627-d6db-422c-a406-91d15c93985c"
    }
   },
   "source": [
    "Among them, we will use 50 classes that we are interested in (i.e. the actions make sense to demonstrate in front of the webcam) and ignore other classes by filtering out from the model outputs. This will help us reduce the noise during prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "nbpresent": {
     "id": "cb834095-db67-4b4b-87c3-a133a848a484"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TARGET_LABELS = [\n",
    "    \"assembling computer\",\n",
    "    \"applying cream\",\n",
    "    \"brushing teeth\",\n",
    "    \"clapping\",\n",
    "    \"cleaning floor\",\n",
    "    \"cleaning windows\",\n",
    "    \"drinking\",\n",
    "    \"eating burger\",\n",
    "    \"eating chips\",\n",
    "    \"eating doughnuts\",\n",
    "    \"eating hotdog\",\n",
    "    \"eating ice cream\",\n",
    "    \"fixing hair\",\n",
    "    \"hammer throw\",\n",
    "    \"high kick\",\n",
    "    \"jogging\",\n",
    "    \"laughing\",\n",
    "    \"mopping floor\",\n",
    "    \"moving furniture\",\n",
    "    \"opening bottle\",\n",
    "    \"plastering\",\n",
    "    \"punching bag\",\n",
    "    \"punching person (boxing)\",\n",
    "    \"pushing cart\",\n",
    "    \"reading book\",\n",
    "    \"reading newspaper\",\n",
    "    \"rock scissors paper\",\n",
    "    \"running on treadmill\",\n",
    "    \"shaking hands\",\n",
    "    \"shaking head\",\n",
    "    \"side kick\",\n",
    "    \"slapping\",\n",
    "    \"smoking\",\n",
    "    \"sneezing\",\n",
    "    \"spray painting\",\n",
    "    \"spraying\",\n",
    "    \"stretching arm\",\n",
    "    \"stretching leg\",\n",
    "    \"sweeping floor\",\n",
    "    \"swinging legs\",\n",
    "    \"texting\",\n",
    "    \"throwing axe\",\n",
    "    \"throwing ball\",\n",
    "    \"unboxing\",\n",
    "    \"unloading truck\",\n",
    "    \"using computer\",\n",
    "    \"using remote controller (not gaming)\",\n",
    "    \"welding\",\n",
    "    \"writing\",\n",
    "    \"yawning\",\n",
    "]\n",
    "len(TARGET_LABELS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "3b3f708a-26d1-4a0c-bf1c-86fcf842751b"
    }
   },
   "source": [
    "# Action Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## From a video file:\n",
    "Here, we show how to use the model on a video file. We utilize threading so that the inference does not block the video preview. \n",
    "\n",
    "For this example, we'll use the following video:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<video src=\"https://cvbp-secondary.z19.web.core.windows.net/datasets/action_recognition/action_sample_lowRes.mp4\" controls  >\n",
       "      Your browser does not support the <code>video</code> element.\n",
       "    </video>"
      ],
      "text/plain": [
       "<IPython.core.display.Video object>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Video(sample_video_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the video to our data folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "210892"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get(sample_video_url)\n",
    "open(video_fpath, 'wb').write(r.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we already have our `learner` setup, we'll pass the video to the `predict_video()` function to get the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total frames = 702\n"
     ]
    },
    {
     "data": {
      "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCACGAPADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDtzZ3JtlurSbZeSkSTPn/WcdMdKlu5Le7tA0UiSSYzFhur9q0zHiMqgxhcCqdnY/ZrOKOcQl0OQUTaK4HK56Ki9jMaxSWxDasyLIX5IfAGegzSXM0WnkiaOMQSL8p+80jehrQYi+lljmtQ1srApIWBDke3tVS5EEd5GLxo2DnFumz7p70cxDRnQKyaOkm6Vo0JYx+UAWGegH+c1WubeWSee9hRbViqbJnOGwByNvQVvxRyqjea6vzxgY4rNvROl/G8dn5yMCrEvwB9DxQpCa0KUljMkUdxZybZWbzJuc+Zn9KddGK4t1lt5I5JesHz8Fq1XQ+SyqADtIUelUrez8m1iSYRM6d0TaBVcz6kWM19Pia2ifUnAk35LB8Ak9B2pLmaGyDrNEqwMuEAGd596vcXc0kM1sGt0IKSMQQx+ntUEywx30Qu/LZWyLdSn3T3/pzTT7ktdilCHGlxys7tGnzOnlBdw9MVHJp63Es05t9itjZK77cDH93titmGOZS/mujru+XaOAKp33mpdxvFavOp+UqG4I989KV73Kj7tmS2lwltbDcwBxwKz9Q1i9vYygkGnWij5rgjczeyist/MsbqbT5S5jjfMTn+JTyK0njtFsQ1xMArD7ua5XGzPTp1OZXMM6nd2l35lnfnUQW5ikj2kD1HvV291l57bfKdrN/CRgiqJtLNSWhmAH1rFu53mu1gibdk7UA9a0tcVSoza0GJJL+bUJ43kigO4lVztzwDXQtbT+bPeW0XlfMGWSRvvLjpg1d8O6WdM0tImAEsmHlI7mqmtLIL5Qd4QjgZqpT5Fc5IUfbTtexZW0ubRYZLSU+VgvOHOc5x+nWpbyHz4hPalXuP+WZDZB9f0zWDM8ixgB3IHYmuw8Gt/wASW6YffAY596inX55WsbVsE6NPnvczP7Ji2QS3gPmIOTuIUtTLueKzjeG4hQI3+pVRnd7/AFqew1W+1Kwv472cSbYsgBAuOvpRE6JPDJe7ZDJxbnZkr610bHF6FdEK6TFPM0rwxgF42iALjPpVd7W6R7i9tkWAs4YSSt99cdCp6V0UMc6eZ50gfLZXAxgfSqV+ki3kbCye5RvlYbhjn27UlPoNrQom1ubRLeS1cm35aYSHPXuPQDnpSanFvhWe2CyzkYSPcCrA9eK3JYd1u0YXAKEAenFVLWwSC3g86GFZoxwYlwBS5gsZK6TH5kDzTSI+3aUV8BuOn/1qp34tYbRreSCKB1OIVIzkevHat7y4tQuHjms22W8mVkblWOO3rUEkEDaipvRAZsFbXd6d/wAeRT5hNGZcJGqW0l0YDa8bMRkAP0zVGO2uYLfbHHIsUzESNIfmHPDLiuqjjlaMfaFQvuPTkY7Gsu6wdS8qa3uZkcgqccIR3HtQpDaOzt5Wsrl7a7vQ7SfNEGPalhVr4eZJMCqswKp9xh0qsTcrLcaityrLGNipKgXPPr2HNWJb02kcctzHHDCw+ZlfPzdgOO9Q4u2hcdB1s9p89vbOn7o4ZFP3T9KZ5vmyyIYmXZjDMOufT8qqvewy6fJPbF4OclvJ5Jp7X8dxpE11CzIBGSCy4IOPSiwXT2KzpIusxsLo+W64MB9geaW4v7SCcQyzKrscAGoTLc3flW6OySBQ63Jjxn1+U0t1d2yXccMsTvKBw4jzimTfRle3drOZoLu7DNIcwhzyfUZpY1e7Akkk+VHIwv3WHSmyiRryS6W4VlgQja6DPt3qT7QY4Y5Joookc/MwmBx+FU0yU7aDYXtlZraFhmLqvpmk8wS3Jj8k5j53MOPwpq3NvcQTyWyyI4wGfy92fwFattPa2Fklxdc3JHQ9j9KluxVOm5MiGgJFetcXGoeUZEwYScgH1+tPutNaKNmilhkj2n5t2CPwrj9Z1S7uZc+YQzEAHPSnSSyx/Z1aQyOrBmIPWsud3O36orHBa94kv7TVLmyv4wMPwoPzBe1UY/E0BjKefIg9CK9B8R6RpfjX7RNd4tbuCLCXUS8A9gw/irzLxL4F1nw9bwXUiJcWU/8Aq7i2cOCfRh1BroioTOaSnTJp9bgwdtxn6Vmprohuo5Fi8xUcOQT97HaufyQaN3PWtlSRzOu2fQPhbxDceIlS/kRrdZDs8jJKjHoa19WiilljCKDJtPOelcD4S8dwX1taaTeFLS4jQRxzhQFAFd1e3s1nNb/ZHEsoTG8LkN9K5a8NDpwk7VLmJcYUlSOc4xXUeFLm3sNMuft0y2+XKbZcLyfr3rmrqS4kuXmeLdI3zN8h4+la+lrc6nbSXE9l5sUsoZjIvYDGQD1rloRftG0epjp3o8o2wfTbKK9H9p2zNLFtC+cmT19Ca09PlLHyzEwCqPnPRvpVcaZbSrLs0p4HH3ZPswOfwq5pzmSNonSZHi+V/NXGT612yTseLEbepKmoWs0d35aEhGh7Pz/9erNzd2tmyrPOse48ZrKiubl7SKP97KJCVE0keDG3Yj1Aq5eT21nDCuooblwR84hyM+v/ANapsFxsTyWl3Ibu7Q28zZgyRwfT8qaiy3ryK86qkcmCsfIcehNNuY5rzVYRFcIEiHmbJIM4HTjPenwXcptPtP2KKMO37xlkBAHdqbiCZNAIISbWKRdydY93Izz0pEaOW6dDCQ8XR2X19D+FQR6hYzPcPbMqzhMmUxE5A6fUVJpt/FfwYSQvNGNsny7eaVmVdFa4N1Fq0OJk+yyAKYyBuzzyP0qWWeGKRY5JUV36AnrWcuoPcW8APky3TEiOdUICP6EGp719PgaA6ksTz5B3CM4z600Fzo2t47mdoru53yyx7WgjbCcc5xVGHT5WZ4EjZo458kXPMWBnlQO9V4AYQbxYntfNlURzrmQ49Nv9afq+sSaNJFcahf8AkRMSVjVNzOPTHaq1TsK5cLXmorCYYzbQqT5iycbhxjHt1qHV7u3spY+G8wYbyFOA4ziuGv8A4gtAGg0yBvK2spe4bLHPpj7uMVyOreJNS1GTzZ7yTcBgBTjHeqULivoez3niCxslQTSqiEZUd1+tctrPj3SLclredbyYrhEVMbT65rx+5u5rtgJJXZlGNxPOKjQGL585NNUybna3njfWbuTejQW6gEYROo96qjxlqqxGLzYip65SuUkvpfu7eKjW5GMnrWnLoJPU9U8G6xNrmvCDULhPsqIZXQLtDsOnIrpb+7jupisf3FPG3oPxryTwtdxLfsZGC8YBNepWvlSRAoRgjqK5Kq1PRw/LbUoXyAQE9xVaGSSeMLGnzDuTWvcW4kBXtWe1u9r+8RtuOcms4wOiVRIY8IttPS1gAUklpGHVj71nXPiyPRo2vLq48+RVaOGBWHzMR1I9Kqz64WDNgMqvtauW1qxkuZTIMMdu5GH8Q9PrW8KdjjrYjojjpW3yM2MbiTgdBzSBM9asvFycCmiIjsa7L6HnW1ERAMVuaR4m1bRpIza3jqkfSJjlfyrHC09V5qbFWPR7X4ozTywrqVmdiNv3Wz7GP/1q9C8N+PPDV+kNhbTtbS4yI7gbcn0z0NfPQIBqVHwck9OlRyRK5mfWO44Bx19e1c7rt1HaxTGZmVXYKu04ydv8uK4H4ffEUA/2VrVziFBiG4fLEf7JrtZ7nTtVvbgWSrczGNTufKqMDpzzmolGzC5YGp3KaWLh7CRSOxYcgd6GvIZZY2klNvcLH5ggmxjHrVVwhnFndxXMgkQKCEyIz7Y7VWLWtvZ3enpDM5KZVmTHA7buw+tRYdy2pu57ZZry9jhiWTfHLDwzD057f4VWk0yO2uUFmbkxXKP8kfKkn+97c1WulinubaxisC0iBWKiY4xjP3v61eSa/kjmt7WXc6SlCpTHlL2G7ofrTegJ3HKt1bWYsILNIrnygwaIfuwc9ye/51LcsdP0VfPnMcg4aSIdWP1qtbn7TZSXwvWkuoImTcqlQp56r/FTFNheqd8zXtxOgJjXKKSB12ngHjvStcpFiHVIzpgm+yTEJxjYMEjuD6UxrmK6kilEojkC71t7gbc9gTUMs0S+XZ3k0kaNGFVFUgofQkdap7bS3iu7GSRp59hWN2U7sDnaCelTbsFzoNT1220KwS5mEjIseUGfkLdsV41r/iG81u9a4uZSxY/KgPCD0FbvxL1yW5ntbLzflhXJUDArzt7jOea3jETZotcbRljVGW4Lv14qrJMT3qEsc8VokhGgHQAnoaY1ypHWqRdj1pBmkJll7gY6CqrSFmoIyOtINo61ZJJFJJG6urEEdCK6C18Z65ZlFikjdB2kXOa5ozcEVYtyWxmolBMuM3HY7n/hYl8IV3WsbSd26D8qyr7xXqeptnzvJTGNiGsZ+VquHCtikqaG6kmbWmyNKrW+SSeRz1Ndr4X0m11Cf7Dqc6QRxq0nml1BJP8ACCzACuX8LWf2tLlkwJEC4J5457VsRaZe61d+TY2xOOCCuenXiqUddDNsyvEmhWUGrXENhdpcRxHAlQ/K/wBK5eeHypCjY3elemy6ZZ6JMVvYGubiAgyI3A/CqOoeH7jxncpPpGniGe1b7PO2MIFxlSMDBI74zVTXLuVFOWx50UxRitTVtG1DSLiSC9t3hZHKfMpAJHUg96z9mFBqVJMHFrci5zTZH2occn0qRioHJqvIw3DbzTQixFI0TptOGyG3DtXs/gTXpL7SzKI4pplkEcqxr84H9414hvIUgck12Hw81P8AszXli8x0S7UxttPftUTV0I9Z1Z76IJI0jY87coi4ZVx/ERVm8aWBzBGjywXCGSdic+Wvf8OayrS6lsle5KyzM8u11n+Vdp9Dzk1oahbWbMLuNnEkYBZEbcCD2PpxWNugxII7aCO3js50micsVZD+8GR0H/16tWd8wspFNpdt5HGWUEv/APXqgl/bpo0kkEbwgyZjO3ds461LFNHLMDZ35N3KQ0jMpCAKOflNJoaY+wt3glCWsqTCX5rhZT/qwegH61DCn2W/urwWlu8UQ+9ExLKfRR603Tre6S8urhZkjhZ2y5YEOM9hnis3SjJBexQzXDWy7twSQE72FVbVjuaerXd4LU3IQwxiRTE2P3nI5BFFzNPHFEVje4N2gDcZ8sY56fWqz6hIt7PcS3AuBEMC32ldyk4zzwOtTXttDNbxX0FxLAVXKpuJ49OOn8qWtgTMjx9odh4ijiv9NlMF+IyHhYfK4HTntXj13a3NnN5V1DJFJ6OpGfoe9fRSRJqkkrHyIMx42twR9KgvdL0nV7b7HcbH8v77uv3R659a1TEj50IPpSgjHQ16xffDrRUGYRcQ7n2iRH3hh6irUfw58KW4hLSXlzggyI52/nVBc8hhtZ7rcYIZJAOuxc4rUt/B/iC7lVINIumZuhOBn8K9tt9O0mxdYdOgW3tB97aASa1pN9vpgubExiIOTvPUe3rTJbPDZvhz4ht+JYrZDnvL1+lOg+GWvXMbuptRt5OZK9ae6adIYLsqgjOPMA5B+lWn8myJhV3cSDqBgD3pXE2fN91Y3Gn30ltdRGOWNiCD/SpFbaOK9O+IOgW0+mDVrV0ZoGCPjgkGvMimFpgmSo+9KglUg7hTUkKNhulTMQy8UyjU8J6wmma/bSzLmIuEk5x8pIya+h59VXSbWAadp08qzDzESFO5/iZjwF96+W2G1jkV7r8JvFUuuaM2kXEyiS2TCszYLrzgHuRx+tF7EpJss2ljDqeuajeayY3kEmPJhyEGB0Zj1PvUuoeIzYILPTrQqqDCW8K8/Wp9TS7tI5LCzhjERfK3DcEqeen9aggtrXRoGlMpmuWOZZyu53J9PauStNy3PSwsUloJpVjPqsfm65DGtn2guuSw/pXn/jPwNHpLyXWlXS3FpksYGHzx/T+8BXon2W6vWEl3LFBFjIXduc+1PY23CRRyMV43E8VhGpKJtUhGR84y7t/XikCk16b438FWNpENR07MSMf3kR6L9K89a2YHAyRXfTnzI82pTcWQxxYBLkCpRI3WFDkdGPHNNaFxgEHFTrjA68VbMj2fQ9Ts9Q8Naet3doYzGFlJ4ZJR/e9qv2N1DZziKCRVLuwY3Bwsi/7HFea+BdSgsNTeK6TdFcDau7oG7Gu5jeythL58Jy7DaxOcH19qylETZo21xc3PmXEcKQ2kZLyRYxuI7Y78fhUE97pEEqXVvC7SsDuhZMDn/PapdUmnt7m3mjmRo5PnVFH3yMenWqUSRapdu008VszjO3bgn6Glbqwb7GjaG3tNPM91aKILhwAyEljn1HpU0txbrdlHSeVmUGFzH93A459Kz5Vs7lI7I3MiNACfMYnbjtTLq81COK180kRj/VyRcblx0NKzb0KT6FiG6a80xxeNCqzSYmfcAQMCoYHj02+kSK4WWQMFVZPlj2EZ+9Va7nsZ7uAQ25S1QbWCjGRnnIquDYi6ZnSQ2mOFPbinyhc7BHtbWBjaIZruPIHy/eU+vrVe0jOp+baPJGFJ3gbQGz6EjqKatvfWN0qp5YMvAG4cVFK4vZkjtYgskZ+Zt2Mkd6LpA22SWo+1XSRuHudjEDY33cdzmmotxDK1gyRbpeWbHT0qikkgYuI3JQEbuRtprTS5aXG59uPM5LDNaEXNCWGe8vTE5TzVUKJFO0cdOOn/AOqod0lvcOEO1g2Mg7gf+A1FsdrV7gsmAVGd3z/lUMVw8Th4jtfP3lPP40yW9TWElhFcGW53SPKD5ny52t/n2qrPe3K3UTSFogCVA25yvaoVtZZYJLlJoEcsWIVuT9aWS9lFsBN5Uzyjhhzsx6/nUDuYPjq1S28OTRxvvjyCpzwteUjGFye1emeMy8ehS7yHjfb/AKs8L6V5czYIqhRGzoMZWmQy4OGqTdkYqGRcNkVRoiy0e8Air/hzVptA8Q2uoRkERONyk8FTway4pcDBp5+YYzj3FJ7DR9Kancx38MGprGqxTQrIpXn5vSs+zRzLvYKZ9p5zxGK5DwJ4mtp/Dj6RqDzSPbtmHOSTntXS299Ld2qJHi3jfkqpy7fUjoK45xbZ3UZpIz5LjU7vUHSB4o7ONirSFcs30rT+0+Qnl28DSN3Y1XluBJbiK1ySHweMH361594l8VT6V4iJ025DBI8XEe0FC3p/+qmqVypVkjp/FfiOHS9KdHG+4eIhEPqeM15LDOSAT171Nqet3WtXLzXL4J6KOgqivFdEIWRy1Z8zNATK3DAUjAqfkUMKp5OauxyBYwDWhiPtru6tJ0mhXa6Hcpz3r03QZm13Txe3DLFLgvIQc/NXmAlRmxmtjw5qDafqYxO6wygq6549jik0Kx6HaPLFMiQnB3FcIN2QeuR2q9AdOtt0cAeWdctbvjOTjgfnWXFcTQ4eFypJOTH/ABA/3qspZyw2iXkFxbgLztVu/X86mxKY+GVri5ks7txEk/L5XGD9fwqvdBoLuS2aTzPK4XOeB2q1d3BeFYJ0hlmkAkMwbK4PbP4VnzxtDetGzK2FB+U5XoOhoW49ytNMQxxwPaoGlbpSzH94frURIqyjqpFjeBOSZA/Ifpj/AGfWnXVwt1IjCJE2joeDn1FRwzeSyuUR+vyy/wBKZDH50yJuwD3kPK+4qNyBVvZ1tpIVk/dsMnPB/LvRHdMlu8WASdvzgfOOfSo7hPKldfMV8L9/OW/CkC5jklLFjuAwv+s6dfagCWaCaNA7oQHIw6/eI96W2uPsxk2xJIX4IxyPrULzvNtDyM4Q4AHVfrUW/cij73JPy/1qyXuTQNEWBkUsvJ/d8fnTY52is3hESMrDduA4FV92fmAzgEbh/D9aWaczFS4UkIF3IMD8qRJgeLJJf7AeMEqHkGeetedyivR/GEQTQwyPu+ZTj0zXn7xZz9aRrHYqDBbk1KyqVq7p+g6hqs/l2FpNcSE9I0zj6mu+0z4K69cxq99NDZqRnafnYfgOKYzyw9eKfGTu5PFeyXPwMUQkwau5fHG+Dj9DXn2v+Btc8OyE3NsXtwf+PiPlfx9PxoAy7G7azuVmRiCOvOOK7mPxfoMNgri0uJbhxteLHyp7hs8/lXJ6P4S1zXSf7OspJlB5fGF/M8Vu/wDCrPGUa8aRu+kq5/LNS0Um1sY+s+L9TuZmjsn+xWknSNPvY9Ce9c0V6jPX0ruW+G3ipRtfQLj/AICAarv8OfE4PGg3xPoIqaC7OO8vYN1SlWVVJHDDIrRuPDmuRvsOkXox6wtVxtMnWyWK4tZ48DqyEYNFwMLIp27C0txbSW7lXXjsw6VCGyMVSECuRITVmOcggg8jmqo61Ig60MZ6/ompudPinjWOTzUAKhehp8Hlb/3qswww/dcYP+1Wdo9tJbaNbW8mS+0Ns6Yz71e6kD7xU9BxiosYvckWcrYm28qNlOWLgZC1WH+vPTGwY29KsSTGZUU4OxSvyDGPqKhLL5isrA5TnaMCqKRSk++aYRxUzj5zUbDiqLOidozHF5YYcHd5h5/CoWwcdfu8g/ez7VNsTHSo3VccCub28TT2Ehj8scE9AM/xZpoyCxbnJzkdfqaXAxxnNbGmeFdX1Yho7YxQn/lrP8q/h3NUqqfQToNbuxj7h8vRuc8N0+tXdP0i91aVI7WDzQM5kA2qn1Nd9pngfTrEK95m8nHbGEX6Dv8AjXRCNY4ljSNY414CgYAqua5nyLoeXL4S1YKyvaBjtIVxKMA/TNKngnVmUCQwKcfeL5/lXprBAMMUH1asjUvEOiaTn7dqNtFt6gyDP5UXDlOGufhjLqkDQT6isakfN5aE/wAzVzT/AIX+F9O/eTW8l7KDy077hn6CqerfGrQbNHSwtZr2RehKhUP4nn9K4DVvjT4jvd6Wa29jGehRQWx9cVVxpWPbxJbWFuI4VgtYE/ugIB+NYt78SfDeiZMl4Lluhjgy5J+vSvnTUPEOqaoT9u1G4nz2d+Py6Vm+YQTg07ge4X/x5i3FbDRBgdGmlwfyGaxpfjbqcwIOl2RU8EHJrycuTSb29am4HuugfHC381bfVtM8qL7qyQvuC/UEV7Ppl9Z6pYxXllKs1vIMhh/KvidZDu616x8HfGc2j6u+lzuzWtwhKKTkK4BIqQseweM/G9h4UtWMu17gjCp15+nevBda+J+t6tO6rcyW8RPGzFU/HmsS6l4kuN03mLGduc8Z7/rXIsc1VirGtLrOtSSGQaxdkn/psaI/FHiC2YAancsB2Z9wP4GskO3Y0hd2OPmpDszoh4wuJk8vUdOsrtD1LxbW/NaZnwpfHLRXunSHqUYSr/jWBskPUH8qcI29D+VMLG+fC9ndY/s/XrKQnosxMZ/UVoaN4Lu0vlmumgeKNgQsUgbf9fQVyao/pUirMvKl19xmn8wa8j1v7LPHgLAxA4VR0AojAWVRLGzhWyYzlf1ry+DUtXgP7q9u1/4Ga3NN8VarAVW5lupV9Q3+INDdhezR1pZWkY9RuOFGFKj+tQ7iZTlgTt7DFR2/iOK5UBpGB9J7bcPzFaKNa3YDKYjLjGU+X9DWftl1K9hLoZz/AHjUJrUOlsST5v8A47/9ekOlSdnU0/bQ7lewqdjQpCOKKK4jtNnQ7q305PPWzjkuc/6yQbtv09K3H8bXKjBQ59jRRWsW0S6cZPVFV/Gl62dg2/X/APXWTd+ItcuQQmp+V/uwA/zJooqnJoh049jmtQt9Y1BSJ/EF6ynqqkID+C1zUvhSEOS0zuT1JbrRRWMqklsylTi+hA3he0QEsX/BqgHh2wL8q5+rUUVn7Wd9y1Sh2FfQdPU7RB0/2jSDRLDJ/cCiihVJ9ynSgtkH9i2IP+oX8ql/sqzVM/Zov++aKKr2ku5HJHsLHp1rkD7PH+Vbeh6RZtfxN5CAg5yBiiikpy7l8sewy60q0W4YrbxA5/uio1s4AR+5i/74FFFS5O+5SirEi28PTyY/++BTxbwg/wCqT/vkUUVN2XyrsWktYWjGY1/KmmxizlUQfhRRSTZXKiaG3ikyhUA+oFWltF+6QpH0ooptsSiiwmlo6hlbHGcUo02L+Lr7UUUk2DSJV02JRxUy20Ua5CKD64ooq0KyGsnpTMkUUVRKP//Z\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "7.0 fps<p style='font-size:20px'>opening bottle (0.082)</p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "video = str(data_path()/\"sample_video.mp4\")\n",
    "learner.predict_video(\n",
    "    video,\n",
    "    LABELS,\n",
    "    averaging_size=AVERAGING_SIZE,\n",
    "    score_threshold=SCORE_THRESHOLD,\n",
    "    target_labels=TARGET_LABELS,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## From your webcam:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section we'll run the same model for prediction actions using our webcam. First we'll set up the webcam params."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Webcam settings\n",
    "w_cam = CameraStream(\n",
    "    constraints={\n",
    "        \"facing_mode\": \"user\",\n",
    "        \"audio\": False,\n",
    "        \"video\": {\"width\": 400, \"height\": 400},\n",
    "    },\n",
    "    layout=Layout(width=\"400px\"),\n",
    ")\n",
    "\n",
    "# Image recorder for taking a snapshot\n",
    "w_imrecorder = ImageRecorder(\n",
    "    format=\"jpg\", stream=w_cam, layout=Layout(padding=\"0 0 0 100px\")\n",
    ")\n",
    "\n",
    "# Text widget to show our classification results\n",
    "w_text = HTML(layout=Layout(padding=\"0 0 0 100px\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll create a function that uses the `learner`'s `predict_frames()` function so predict actions from the webcam. (This is the same function the `predict_video()` uses too!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_webcam_frames():\n",
    "    \"\"\" Predict activity by using a pretrained model\n",
    "    \"\"\"\n",
    "    global w_imrecorder, w_text, is_playing\n",
    "    global device, model\n",
    "\n",
    "    # Use deque for sliding window over frames\n",
    "    window = deque()\n",
    "    scores_cache = deque()\n",
    "    scores_sum = np.zeros(len(LABELS))\n",
    "\n",
    "    while is_playing:\n",
    "        try:\n",
    "            # Get the image (RGBA) and convert to RGB\n",
    "            im = Image.open(io.BytesIO(w_imrecorder.image.value)).convert(\"RGB\")\n",
    "            window.append(np.array(im))\n",
    "\n",
    "            # update println func\n",
    "            def update_println(println):\n",
    "                w_text.value = println\n",
    "            \n",
    "            if len(window) == NUM_FRAMES:\n",
    "                learner.predict_frames(\n",
    "                    window,\n",
    "                    scores_cache,\n",
    "                    scores_sum,\n",
    "                    None,\n",
    "                    AVERAGING_SIZE,\n",
    "                    SCORE_THRESHOLD,\n",
    "                    LABELS,\n",
    "                    TARGET_LABELS,\n",
    "                    get_transforms(train=False), \n",
    "                    update_println,\n",
    "                )\n",
    "            else:\n",
    "                w_text.value = \"Preparing...\"\n",
    "        except OSError:\n",
    "            # If im_recorder doesn't have valid image data, skip it.\n",
    "            pass\n",
    "        except BaseException as e:\n",
    "            w_text.value = \"Exception: \" + str(e)\n",
    "            break\n",
    "\n",
    "        # Taking the next snapshot programmatically\n",
    "        w_imrecorder.recording = True\n",
    "        sleep(0.02)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "is_playing = False\n",
    "#  Once prediciton started, hide image recorder widget for faster fps\n",
    "def start(_):\n",
    "    global is_playing\n",
    "    # Make sure this get called only once\n",
    "    if not is_playing:\n",
    "        w_imrecorder.layout.display = \"none\"\n",
    "        is_playing = True\n",
    "        Thread(target=predict_webcam_frames).start()\n",
    "\n",
    "\n",
    "w_imrecorder.image.observe(start, \"value\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "3b2047f3-87a7-47bd-a53c-2fa90540be99"
    }
   },
   "source": [
    "To start inference on webcam stream, click 'capture' button when the stream is started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(CameraStream(constraints={'facing_mode': 'user', 'audio': False, 'video': {'width': 400, 'heigh…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "HBox([w_cam, w_imrecorder, w_text])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "acf10288-9c35-45ec-8b20-cbb3782dead3"
    }
   },
   "source": [
    "## Stop Webcam and clean-up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stop the webcam by running the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "nbpresent": {
     "id": "a218abeb-4dd9-4030-9e54-550415ae8a89"
    }
   },
   "outputs": [],
   "source": [
    "is_playing = False\n",
    "Widget.close_all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learn more about fine-tuning action recognition models in our next [01_training_introduction.ipynb](01_training_introduction.ipynb) notebook."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (cv)",
   "language": "python",
   "name": "cv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "metadata": {
     "collapsed": false
    },
    "source": []
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
